Querying Twitter API Data
System Investigation Project
Hannah Luebbering
May 20, 2022
Overview: The following project aims to programmatically retrieve and analyze all of Elon Musk’s tweets from the Twitter API. We compile over 10,000 of Musk’s tweets into a comprehensive dataset and perform sentiment analysis for each tweet object.
| 18.92 K | 118 | 102.8 M |
| Tweets | Following | Followers |
Introduction
Elon Musk, who transformed the electric car industry and accelerated the world’s space exploration efforts, provides the public an unfiltered look into his eccentric mind through his online Twitter presence. With more than 77 million followers, Musk is one of the most followed Twitter accounts, with each of his tweets getting thousands of shares, likes, and comments. Musk tweets frequently every day, covering a wide range of topics from technical ideas to lighthearted memes. Some of his tweets have a significant impact, making headlines, instigating controversy, and sometimes pushing the needle on everything from Tesla’s stock price to cryptocurrency markets. The companies that Musk runs are also hugely influential and disruptive.
To better understand Musk’s Twitter profile, we programmatically
retrieve and analyze all of Elon Musk’s tweets using the
rtweet package, which provides a convenient interface
between the Twitter API and R code. We compiled over 10,000 of his
tweets into a comprehensive dataset. Once we scraped a decade worth of
Elon Musk tweets, we analyzed the data and sorted the information to
answer the following questions.
(tweet types)What is the ratio of mentions, replies, retweets, quotes, and organic tweets?(user engagement)How does Elon Musk engage with other twitter users?(twitter activity)Are there any trends of when Musk tweets?(topics)What are Elon Musk’s most tweeted topics?
The first question examines Musk’s tweet distribution, which we answer by sorting each tweet into categories (based on tweet types) and ranking each category based on the volume of tweets. Analyzing Musk’s nature of engagement, we answer the second question by unpacking tweets containing conversations with and directed to other users. To answer the third question, we investigate Musk’s Twitter activity by sorting his feed into different timeframes based on the publish date. Dissecting the topics that predominate his profile feed, we answer the fourth question by examining Musk’s most frequently used hashtags and words.
Data Description
Using the Twitter API, we compile over 10,000 of Musk’s tweets into a
comprehensive dataset. The dataset contains all of Elon Musk’s tweets
from 2015 to 2022, stored in RDS format, where each tweet is in its
separate row object. All Tweets are collected, parsed, and plotted using
rtweet in R.
| 1 | status_id | 14 | hashtags | 27 | quoted_followers_count | 40 | retweet_location |
| 2 | created_at | 15 | symbols | 28 | quoted_location | 41 | retweet_description |
| 3 | user_id | 16 | media_expanded_url | 29 | quoted_description | 42 | retweet_verified |
| 4 | screen_name | 17 | media_type | 30 | quoted_verified | 43 | name |
| 5 | text | 18 | mentions_screen_name | 31 | retweet_status_id | 44 | location |
| 6 | source | 19 | quoted_status_id | 32 | retweet_text | 45 | description |
| 7 | reply_to_screen_name | 20 | quoted_text | 33 | retweet_created_at | 46 | followers_count |
| 8 | is_quote | 21 | quoted_created_at | 34 | retweet_source | 47 | friends_count |
| 9 | is_retweet | 22 | quoted_source | 35 | retweet_favorite_count | 48 | statuses_count |
| 10 | favorite_count | 23 | quoted_favorite_count | 36 | retweet_retweet_count | 49 | account_created_at |
| 11 | retweet_count | 24 | quoted_retweet_count | 37 | retweet_user_id | 50 | verified |
| 12 | quote_count | 25 | quoted_user_id | 38 | retweet_screen_name | ||
| 13 | reply_count | 26 | quoted_screen_name | 39 | retweet_followers_count |
Twitter API
To get the data for the Twitter account, we must first register a developer account and create an application with the necessary credentials to access the Twitter API. After setting up the application, we generate an authentication key and use the following commands to create a Twitter token that allows access to Twitter data.
library(rtweet) # load rtweet package
twitter_token <- create_token(app = "twitter_app", consumer_key = "api_key", consumer_secret = "api_secret", access_token = "access_token", access_secret = "access_secret")
use_oauth_token(twitter_token) # authenticate via web browserSearching Twitter’s full archive API with the rtweet
package, we run the search_fullarchive() function to access
and extract the complete historical information of all the tweets
associated with a particular user. We can then save these excerpted
tweets to an RDS file. The sample code below captures Elon Musk’s tweets
from January 01, 2010, to May 28, 2022.
df <- search_fullarchive(q = "from:elonmusk", n = 10000, env_name = enviroment_name, fromDate = "201001010000", toDate = "202205280000")| created_at | screen_name | text | favorite_count | retweet_count | quote_count | reply_count | is_quote | is_retweet |
|---|---|---|---|---|---|---|---|---|
| 2022-08-04 19:23:21 | elonmusk | Livestream the 2022 Annual Tesla Shareholder Meeting today at 4:30pm CT https://t.co/KttPYLPcxi https://t.co/eF9kRVfFOr | 0 | 0 | 0 | 0 | FALSE | TRUE |
| 2022-08-04 18:04:12 | elonmusk | (garyblack00?) I had more kids in Q2 than they made cars! | 58736 | 3593 | 1115 | 3173 | FALSE | FALSE |
| 2022-08-04 16:24:08 | elonmusk | (dogeofficialceo?) (chicago_glenn?) (mySA?) (happydad?) Stone of Destiny | 2935 | 178 | 30 | 436 | FALSE | FALSE |
Storing Data in Databricks
Optionally, we can load the Twitter API data into a data management system, such as Azure Databricks, and write queries to run a SQL job and retrieve the data. From the Azure portal, we create and launch a Databricks workspace, establish a Spark cluster, and configure a notebook on the cluster. In the notebook, we use SparkR to read the dataset into a Spark DataFrame and run a SQL job to query the data.
Azure Databricks is an analytics platform based on
Microsoft Azure cloud services, enabling the latest versions of Apache
Spark: an open-source engine providing large-scale APIs in
general-purpose programming languages such as Scala, Python, and R.
Specifically, Databricks provides a cloud-based interactive workspace
with fully managed Spark clusters, allowing users to quickly execute
Spark code in a easy-to-use environment.
Querying Twitter Data
When working with Twitter data, one of the first steps is distinguishing organic, user-written tweets from other tweet categories: retweets, replies, mentions, and quotes. By interpreting the tweet type ratio, we can glimpse into the general overview of a user’s Twitter feed and account type.
Query 1. Tweet Types
- What is the ratio of tweet types: mentions, replies, retweets, quotes, and organic tweets?
The different types of tweets that exist are general tweets, mentions, replies, retweets, and quotes. General tweets are original Twitter posts that include text, photos, GIFs, and videos but do not include mentions, replies, retweets, or quotes. Both mentions and replies are tweets containing other account usernames, though replies are specific tweets sent directly to another user’s tweets. Lastly, retweets and quotes are re-postings of another person’s tweets, although quotes allow users to post another tweet with their own added comment.
As a first step, we distinguish between organic tweets, retweets, and
replies. With the data collected through the Twitter API, we can use
specific columns such as is_retweet and
reply_to_status_id to determine the tweet type. The
following command removes retweets and replies from the data to keep
only organic/generic tweets.
# Remove retweets and replies
dfGeneral <- df[df$is_retweet == FALSE,] %>% subset(is.na(reply_to_status_id))Similarly, we want to create a different dataset for each data type.
dfMention <- subset(df, !is.na(df$mentions_user_id))
dfReply <- subset(df, !is.na(df$reply_to_status_id))
dfRtweet <- df[df$is_retweet == TRUE,]
dfQuote <- df[df$is_quote == TRUE,]In the above, we subset the tweets into five datasets containing only
general tweets, mentions, replies, retweets, or quotes. We then count
the number of observations for each dataset using the
nrow() function and store the information in a separate
data frame containing the tweet types and their corresponding
counts.
Now, for example, we can show information for each of Musk’s retweets and query the data to obtain his most frequently retweeted users. To identify the most frequently retweeted users, we use tidyr tools to unnest, count, and sort each user from Musk’s retweets.
Query 2. Mentions
- How does Elon Musk engage with other twitter users?
To track Elon Musk’s engagement with people on Twitter, we want to look into tweets containing conversations with and directed to other users. We begin by unpacking information for each of Musk’s tweets that mention another person’s username. Specifically, mentions are a type of tweet containing other account usernames, preceded by the “@” symbol.
| created_at | mentions_user_id | mentions_screen_name | text |
|---|---|---|---|
| 2022-07-27 01:40:54 | x924966648922157056 x593977467 x42555311 x16831353 | chicago_glenn michaelsiconolf KirstenGrind EmilyGlazer | (chicago_glenn?) (michaelsiconolf?) (KirstenGrind?) (EmilyGlazer?) 99% of journalism is reading someone else’s story on the Internet, changing it up a little & pressing send |
| 2022-08-01 17:28:24 | x1492510878323081216 | spideycyp_155 | (spideycyp_155?) So much water under the bridge since then |
| 2022-08-04 16:19:18 | x924966648922157056 x9830752 x38672574 | chicago_glenn mySA happydad | (chicago_glenn?) (mySA?) (happydad?) Yeah, pretty good. I like the name. |
As shown above, tweets can mention more than one user in the body of the text. Therefore, we have to process the data so that each user mentioned in the tweet is recorded in a separate row so that we can count the total number of times Musk mentioned a unique user.
dfMentions %>% tibble(user = str_extract_all(text, "@\\w+")) %>%
tidyr::unnest_longer(user) %>% dplyr::count(user, sort = TRUE)The above command uses the str_extract_all() function to
extract the mentioned users for each tweet and
unnest_longer() to transform the nested lists into tidy
rows so that each row contains only one user. Lastly, we count the total
number of observations for each unique user.
| screen_name | n | mention_time | mention_time2 |
|---|---|---|---|
| spacex | 1323 | ||
| tesla | 1050 | ||
| erdayastronaut | 705 | ||
| ppathole | 474 | ||
| flcnhvy | 470 | ||
| teslaownerssv | 459 | ||
| wholemarsblog | 384 | ||
| teslarati | 351 | ||
| nasa | 264 | ||
| cleantechnica | 217 |
Linking conversations together, a reply is a type of tweet sent directly to another user’s tweet. Analogous to mentions, replies allow users to direct tweets toward other Twitter users and interact in conversations. Continuing the same general procedure above, we get the following results.
Query 3. Twitter Activity
- Are there any trends of when Elon Musk tweets?
Here, we provide an overview of Musk’s Twitter activity by analyzing
the frequency of tweets by timeframes such as year, month, weekday,
hour, and time of day. Parsing the information from the
created_at column, we extract timestamps to display the
year, month, day, and hour of each tweet’s posting date.
df$created_at <- to_timestamp(df$created_at)
df$year <- year(df$created_at)
df$month <- date_format(to_date(df$created_at), "MMMM")
df$weekday <- date_format(to_date(df$created_at), "EEEE")| created_at | year | month | weekday | time |
|---|---|---|---|---|
| 2022-02-25 11:02:23 | 2022 | February | Friday | morning |
| 2022-02-22 10:21:06 | 2022 | February | Tuesday | morning |
| 2020-09-28 07:46:13 | 2020 | September | Monday | morning |
| 2020-07-14 18:42:01 | 2020 | July | Tuesday | evening |
| 2020-04-23 03:29:23 | 2020 | April | Thursday | night |
| 2019-10-11 17:35:59 | 2019 | October | Friday | evening |
The above data allows us to explore Musk’s Twitter usage, ranging from the frequency of tweets over years to which days of the week or hours of the day have more or less activity. As a result, we see that Musk is most active on Thursdays and Fridays, mostly tweeting at night.
We can go even deeper and look at the exact time and day of the week in which Musk has the most activity posting tweets with a detailed histogram. Consistent with our previous results, we see the most activity on Thursdays at 6 pm.
Query 4. Tweet Topics
- What are Elon Musk’s most tweeted topics?
Continuing to analyze Musk’s habits on Twitter, we want to know which topics dominate Musk’s Twitter feed. To get such topics and themes of the user content, we examine the most frequently used hashtags and words for all of Musk’s tweets. Extracting unique hashtags and words from a tweet requires text mining tasks.
Hashtags:
We first extract hashtags, all words preceded with a #
character, from the content of Musk’s tweets. The following command
cleanses and unpacks the text of each tweet to get a string array
containing only the hashtags. Then, we calculate the frequency of each
unique hashtag.
hashtag <- df$text %>% str_extract_all("#[A-Za-z0-9_]+")
hashtag_word <- unlist(hashtag)
hashtag_word <- tolower(hashtag_word)
hashtag_word <- gsub("[[:punct:]ー]", "", hashtag_word)
Topic Words:
Next, let’s look at the words Musk mentioned the most in his tweets. We first clean up the text in the data by removing punctuation, URL links, and symbols, and use tools to convert the text into a tidy format. Then, we split the text into individual words and remove stop words.
# Regex for parsing tweets
replace_reg <- "https?://[^\\s]+|&|<|>|\bRT\\b|^<"
# Clean text
words <- dfGeneral %>%
dplyr::mutate(
text = str_remove_all(text, replace_reg),
text = str_remove_all(text, "[[:punct:]]"),
text = str_remove_all(text, "[[:digit:]]")) %>%
# Split into words
unnest_tokens(word, text, token = "tweets") %>%
# Remove stop words
anti_join(stop_words, by = "word")In the above command, the pattern matching function
str_remove_all() removes unwanted text, and the
unnest_tokens() function splits the text of each tweet into
tokens, using a one-word-per-row format. We then use the
str_detect() function to filter out words by removing stop
words, unicode characters, and whitespace.
Above, we used the unnest_tokens function to tokenize by
word; however, we can also use these functions to tokenize into
consecutive sequences of words, called n-grams. We do this by adding the
option token = "ngrams" and setting \(n\) to the number of words. Setting n to
\(2\) allows us to examine pairs of two
consecutive words, often called bigrams.
bigrams <- dfGeneral %>%
dplyr::mutate(text = str_replace_all(text, replace_reg, "")) %>%
# split into word pairs
unnest_tokens(bigram, text, token = "ngrams", n = 2) %>%
separate(bigram, into = c("first","second"), sep = " ", remove = FALSE) %>%
# remove stop words
anti_join(stop_words, by = c("first" = "word")) %>%
anti_join(stop_words, by = c("second" = "word")) %>%
filter(str_detect(first, "[a-z]") & str_detect(second, "[a-z]"))| name | freq | time_posted |
|---|---|---|
| tesla model | 35 | |
| boring company | 28 | |
| falcon heavy | 26 | |
| space station | 25 | |
| cape canaveral | 22 | |
| climate change | 19 | |
| tesla team | 19 | |
| tesla owners | 18 | |
| static fire | 16 | |
| giga berlin | 15 |
Sentiment Analysis
Here we use the syuzhet R package to iterate over a
vector of strings consisting of the text from all of Elon Musk’s tweets
in our dataset. To obtain the vector of tweet text, the plain_tweets()
function from the rtweet package is used to clean up the
tweets character vector to cleaned up, plain text. We then pass this
vector to the get_sentiment() function, which consequently returns the
sentiment values based on the custom sentiment dictionary developed from
a collection of human coded sentences.
round_time <- function(x, secs)
as.POSIXct(hms::round_hms(x, secs))
sent_scores <- function(x)
syuzhet::get_sentiment(plain_tweets(x)) - .5
df.sentiment <- gfg_data %>%
dplyr::mutate(days = round_time(created_at, 60 * 60 * 24),
sentiment = sent_scores(text)) %>%
dplyr::group_by(days) %>%
dplyr::summarise(sentiment = sum(sentiment, na.rm = TRUE))Extending the above sentiment analysis, the next step is to
understand the opinion or emotion in the text. First, we must clean the
text from our dataset so that it’s in a tidy format. We accomplish this
using the R function gsub() to replace unwanted text and
the get_nrc_sentiment() function to get the emotions and
valences from the NRC sentiment dictionary for each word from all of
Musk’s tweet.
R Code:
txt <- c("rt|RT", "http\\w+", "<.*?>", "@\\w+", "[[:punct:]]", "\r?\n|\r", "[[:digit:]]", "[ |\t]{2,}", "^ ", " $")
cleanTweet <- as.vector(df$text)
cleanTweet <- grep::gsub(txt, "", cleanTweet)
textSentiment <- syuzhet::get_nrc_sentiment(cleanTweet)
nrc_sentiment <- cbind(df, textSentiment) %>%
dplyr::select(created_at, anger, anticipation, disgust, fear,
joy, sadness, surprise, trust, negative, positive)In the above command, the gsub function replaces all occurrences of
the given patterns and the get_nrc_sentiment function calculates the
presence of eight different emotions and their corresponding valence.
The resulting columns include the eight emotions disgust,
fear, joy, sadness,
surprise, trust and their respective
positive or negative valence.
Conclusion
In this article, I aimed to show how to extract and analyse tweets using the free-to-use programming software R. I hope you found this guide helpful to build your own Twitter Analytics Report that includes:
Showing which tweets worked best and which didn’t The ratio of organic tweets/replies/retweets, the time of tweet publication and the platforms from which tweets are published. These are all insights regarding the tweeting behaviour. The most frequent words used in the tweets, hashtags, from which accounts most retweets originate and a sentiment analysis capturing the tone of the tweets. These are all insights on the content of the tweets.